Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-6067][CH][Minor] Compile Spark-3.5 ut with backends-clickhouse #6114

Merged
merged 2 commits into from
Jun 17, 2024
Merged

[GLUTEN-6067][CH][Minor] Compile Spark-3.5 ut with backends-clickhouse #6114

merged 2 commits into from
Jun 17, 2024

Conversation

baibaichen
Copy link
Contributor

@baibaichen baibaichen commented Jun 17, 2024 •
edited
Loading

What changes were proposed in this pull request?

GlutenV1WriteCommandSuite and GlutenInsertSuite failed to compile with backends-clickhouse profiler. I refactor them in this PR, so we can compile spark-ut with spark-3.5 and backends-clickhouse profile. Once merged, we can add compile step in CICD asap to avoid regression.

How to do

I introduce GlutenColumnarWriteTestSupport trait, and use build-helper-maven-plugin to add diffierent implementation per backends

(Fixes: #6067)

How was this patch tested?

Existed UTs

@github-actions GitHub Actions
Copy link

#6067

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen mentioned this pull request Jun 17, 2024
Open
17 tasks
@zhouyuan
Copy link
Contributor

CC @JkSelf, please help to take a look

@baibaichen baibaichen force-pushed the feature/spark_35_build branch from 5834910 to 5ca17fc Compare June 17, 2024 09:58
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

JkSelf
JkSelf approved these changes Jun 17, 2024
View reviewed changes
Copy link
Contributor

@JkSelf JkSelf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks.

@baibaichen baibaichen merged commit 5b87efa into apache:main Jun 17, 2024
38 checks passed
@baibaichen baibaichen deleted the feature/spark_35_build branch June 17, 2024 13:39
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_06_17_2024_time.csv log/native_master_06_16_2024_a08a57c61_time.csv difference percentage
q1 34.50 37.13 2.635 107.64%
q2 23.69 24.00 0.302 101.27%
q3 39.94 37.23 -2.716 93.20%
q4 32.62 32.75 0.124 100.38%
q5 67.62 71.56 3.945 105.83%
q6 6.63 8.16 1.528 123.04%
q7 81.08 80.98 -0.094 99.88%
q8 85.44 86.78 1.338 101.57%
q9 120.83 121.57 0.744 100.62%
q10 46.34 44.17 -2.174 95.31%
q11 19.96 21.42 1.455 107.29%
q12 25.57 25.32 -0.250 99.02%
q13 38.36 39.84 1.478 103.85%
q14 19.32 19.23 -0.091 99.53%
q15 32.79 33.05 0.263 100.80%
q16 13.93 14.54 0.604 104.33%
q17 103.20 102.23 -0.963 99.07%
q18 147.38 142.93 -4.448 96.98%
q19 14.12 15.44 1.320 109.35%
q20 28.78 27.99 -0.786 97.27%
q21 258.28 261.41 3.128 101.21%
q22 12.23 12.41 0.188 101.53%
total 1252.62 1260.15 7.530 100.60%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JkSelf JkSelf JkSelf approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[CH] Support CH backend with Spark 3.5.x
4 participants
@baibaichen @zhouyuan @GlutenPerfBot @JkSelf